feat(harbor): task.yaml/experiment.yaml → Harbor export + coder-eval as a Harbor agent - #166

…ng /app

Converting a real in-tree task (hello_date_smoke_test) through the export
pipeline surfaced a live bug in the bare-`docker.image` branch of
`_write_environment`: it defaulted `workdir` to the hardcoded
`DEFAULT_WORKDIR = "/app"` whenever `sandbox.docker.working_dir` was unset,
regardless of the image's actual WORKDIR.

`coder-eval-agent:latest`'s real WORKDIR is `/work`. Harbor's shared verifier
runs `docker exec -w <workdir> <container> ...`, and confirmed live that
`docker exec -w` -- unlike `docker run -w`, which auto-creates the directory
-- hard-fails to chdir into a path that doesn't already exist in the image
("no such file or directory"), surfacing as agent exit 127 with no verifier
output at all. The dockerfile_path branch never had this bug (it appends a
WORKDIR line and the directory exists once Docker builds it); the bare-image
branch had no Dockerfile to fix and no way to introspect the image, so it
just guessed.

Fix: `_inspect_image_workdir()` runs `docker image inspect <image> --format
'{{.Config.WorkingDir}}'` (best-effort, 30s timeout, returns None on any
failure) and the bare-image branch now tries, in order: explicit
`working_dir` -> inspected WORKDIR -> `DEFAULT_WORKDIR`, warning
(non-blocking) only on the last fallback.

Re-verified live after the fix: `harbor run -p <dir> -a oracle` against the
converted hello_date task scored reward: 1.0.

Test hygiene: added an autouse fixture stubbing `_inspect_image_workdir`
across `test_harbor_packager.py` so packager unit tests never depend on
what happens to be cached in the local docker daemon -- a real image named
`byod-custom-image:0.1.0` (this file's own placeholder name), built by an
unrelated docker-integration test elsewhere in the suite, silently flipped
a DEFAULT_WORKDIR assertion the first time this ran locally. New tests
(TestPrebuiltImageWorkdirInspection) cover the inspection helper directly
and its three-way fallback order.

Full narrative in tmp/harborframework.md (gitignored), including a scoping
note for the next piece (coder-eval as Harbor's *agent*, not just its
verifier) -- not started, planning only.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Extends the single-task Harbor exporter to experiment.yaml: each resolved
(task, variant, replicate[, dataset row]) combination gets its own Harbor
directory, since Harbor's task.toml has no variant concept. Reuses
resolve_all_tasks (the same pipeline `run -e` uses) for resolution, and
skips (rather than silently drops) a variant whose agent/simulation
override Harbor's verifier-only export cannot honor, detected precisely
via config_lineage so a task's own agent config never false-triggers it.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Adopts ATIF (Agent Trajectory Interchange Format) as coder-eval's
trajectory interchange layer so a Harbor trial can run coder-eval itself
as the agent, not just as the grader.

- harbor/atif_models.py + atif_emit.py: vendored ATIF schema + a pure
  EvaluationResult -> Trajectory converter, opt-in via `--format harbor`
  on `execute`/`run` (writes a trajectory.json sibling to task.json;
  never changes default output).
- harbor/atif_hydrate.py: the reverse direction (Trajectory -> TurnRecords)
  so `evaluate --format harbor --trajectory <path>` can grade
  trajectory-dependent criteria against an ATIF file that didn't
  originate from an in-process run.
- harbor/agent.py: CoderEvalAgent(BaseInstalledAgent) — a real Harbor
  agent that shells out to `coder-eval execute --format harbor` inside
  the container and lets Harbor discover the resulting trajectory.json
  via `populate_context_post_run`, matching Harbor's own ClaudeCode
  agent's exact convention. Verified end-to-end against a real
  harbor==0.22.0 install: a full export -> docker build -> `harbor run`
  -> grade round trip produces correct rewards for both a passing and a
  deliberately failing criterion.
- packager.py: bakes a criteria-free agent-phase task.yaml into the
  image at a fixed path (agent_paths.py) via a Dockerfile COPY line, so
  CoderEvalAgent always has a task to execute without ever seeing the
  real success_criteria.
- run_command.py: when `--format harbor` writes exactly one trajectory
  (the shape a CoderEvalAgent invocation always produces), it is also
  copied to <run_dir>/trajectory.json so a fixed-path caller (Harbor's
  agent log dir) can find it without knowing coder-eval's internal
  variant/task/replicate nesting.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

…plate sources

Several fixes to the Harbor export/agent pipeline, each verified against a
real `harbor==0.22.0` install with real Docker and (where relevant) real
Bedrock credentials:

- CoderEvalAgent now passes --workspace-dir "$(pwd)" to `coder-eval execute`,
  and BatchRunConfig/run_batch/run+execute CLIs gained a matching
  --workspace-dir flag (Orchestrator already supported this internally for
  DockerRunner). Without it, the agent phase wrote to a throwaway tempdir the
  verifier phase never looked at, silently scoring every criterion 0.
- packager.py now emits [environment.env]/[verifier.env] in task.toml, derived
  from the resolved task's own sandbox.docker.env_passthrough (the same
  allowlist DockerRunner already uses) rather than requiring the operator to
  hand-edit task.toml. HOME is excluded (that entry only makes sense paired
  with DockerRunner's ~/.claude bind-mount). Values are `${VAR:-}` templates
  (empty default) so Harbor doesn't hard-require every var to be present.
- environment/Dockerfile is now always written (synthesized from
  sandbox.docker.image when no dockerfile_path is set), so the CoderEvalAgent
  embed always has somewhere to COPY task.yaml into.
- instruction.md is now a fixed placeholder pointing at environment/task.yaml
  and the CoderEvalAgent agent, instead of leaking the real prompt to a
  stock Harbor agent that isn't the one this export is built for.
- environment/task.yaml's success_criteria is now a real `[]` instead of a
  placeholder criterion, now that TaskDefinition.validate_success_criteria
  (the "at least one criterion" check) has been removed.
- TemplateDirSource directories are now copied into
  environment/templates/<n>-<name>/ with their path rewritten to the
  in-container copy; environment/task.yaml's sandbox block is field-merged
  (driver forced to tempdir) instead of being replaced wholesale, so
  python.env_packages/limits/etc. survive.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Installs harbor==0.22.0 alongside coder-eval in the same venv, builds the
coder-eval-agent and byod-custom-image Docker images, then exports and runs
four scenarios via `hb run -a coder_eval.harbor.agent:CoderEvalAgent`:
baseline (plain docker driver, default image), llm_judge (real judge call
inside the verifier phase), docker_custom_image (reuses tasks/byod_smoke_test.yaml's
dockerfile_path + custom image), and template_sources (TemplateDirSource
copy-in). Each scenario asserts reward.json == 1.0, a real ATIF
trajectory.json, and both agent- and verifier-side task.json.

Verified locally end-to-end against real Docker + real Bedrock credentials
before committing (all four scenarios: reward 1.0, no exceptions).

Triggered on workflow_dispatch/schedule/push-to-main only (not pull_request)
so it's informational rather than a required PR check for now — it depends
on live model calls and Docker builds, and Harbor's own release cadence is
independent of this repo's.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Two more CI-only failures found running PR #166's full Quality
Gate/Windows Smoke Test jobs (masked previously by the typecheck failure
that aborted before the test step ran):

- test_execute_format_harbor.py / test_evaluate_format_harbor.py assert a
  plain substring ("Unsupported --format", "--trajectory") against a raised
  Click/Typer error's rendered output. CI renders that error box with color,
  which splits "--format"/"--trajectory" across ANSI escape sequences and
  silently breaks a naive substring check -- reproduced locally with
  FORCE_COLOR=1. Fixed with click.unstyle(result.output) before asserting.
- Two harbor packager tests assert the exported test.sh's chmod executable
  bit, which is meaningless on NTFS (Windows Smoke Test runs on Windows).
  Skipped/guarded the same way test_sandbox.py already does for other
  POSIX-only behavior.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Fixes surfaced by /coder-eval-code-review-full over two iterations:

- Refuse a task with empty success_criteria under `run`/`evaluate`
  (grade=True) -- it previously loaded and would silently score
  FinalStatus.SUCCESS at weighted_score 0.0 since
  validate_success_criteria was removed to let the criteria-free
  Harbor agent-phase task.yaml round-trip through `execute`.
  `execute` (grade=False) still accepts it. Scoped to the
  post-`--resume` task set so an already-finalized row is never
  blocked.
- Refuse a path-traversal-shaped variant_id/task_id/row_id in
  experiment export (`_out_subdir`) instead of silently writing
  outside `-o`'s output directory.
- Convert `--workspace-dir` misuse (docker driver, >1 task) from an
  unhandled ValueError/traceback into a clean typer.BadParameter.
- Escalate a missing `template_sources` directory during export from
  a warning to a hard failure -- the agent-phase task.yaml still
  references it, so a silently-skipped copy shipped an export with no
  starter code.
- Add a `harbor` optional extra (pinned harbor==0.22.0) so
  harbor/agent.py type-checks against real types instead of scoped
  pyright ignores; wire it into the two CI jobs that run pyright and
  simplify harbor-e2e.yml's install step to use it.
- New test coverage: harbor/agent.py's run()/populate_context_post_run
  (previously zero coverage), the empty-criteria and workspace_dir
  guards, the path-traversal refusal, and the template-source warn vs.
  hard-fail branches.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

…utput

CI renders Click's error box with color, splitting "--workspace-dir"
across escape sequences and breaking the new
test_workspace_dir_with_docker_driver_is_a_clean_cli_error test that
raw-substring-matched it. Same fix as the existing --format/--trajectory
tests: assert against click.unstyle(result.output) instead.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Lets the same per-task agent image double as a Harbor agent image
(coder_eval.harbor.agent:CoderEvalAgent, a
harbor.agents.installed.base.BaseInstalledAgent subclass) without a
separate build, matching the harbor extra already wired into both
pyright CI jobs.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

…ment_export